Image processing apparatus, image processing method, and program

ABSTRACT

An image processing apparatus includes a special light image acquisition unit that acquires a special light image having information in a specific wavelength band, a generation unit that generates depth information at a predetermined position in a living body using the special light image, and a detection unit that detects a predetermined region using the depth information. The image processing apparatus further includes a normal light image acquisition unit that acquires a normal light image having information in a wavelength band of white light. The specific wavelength band is, for example, infrared light. The generation unit calculates a difference in a depth direction between the special light image and the normal light image to generate depth information at the predetermined position. The detection unit detects a position in which the depth information is a predetermined threshold or more as a bleeding point. The present technology is applicable to an endoscope.

TECHNICAL FIELD

The present technology relates to an image processing apparatus, animage processing method, and a program. Specifically, the presenttechnology relates to an image processing apparatus, an image processingmethod, and a program suitably applicable to an apparatus such as anendoscope that captures an image of the inside of the body cavity of asubject to obtain information of the inside of the body cavity.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of Japanese Priority PatentApplication JP 2014-229368 filed on Nov. 12, 2014, the entire contentsof which are incorporated herein by reference.

BACKGROUND ART

There has been widely used a frame sequential endoscope system thatsequentially applies beams of light R1, G1, and B1 of three colors to atissue inside the body cavity using a rotary filter and performsdiagnosis using an image (normal light image) created from reflectedlight images of these light beams. Further, there has been proposed anendoscope system that sequentially applies two kinds of narrow bandlight beams G2 and B2 having characteristics different from thecharacteristics of the above three-color light beams to a tissue insidethe body cavity and performs diagnosis using a narrow band light imagecreated from reflected light images of these light beams (e.g., PTL 1).

When diagnosis is performed using an endoscope system that acquires anarrow band light image, for example, a lesion such as squamous cellcarcinoma, which is difficult to visually recognize in a normal lightimage, is visualized as a brown region different from a normal part.Thus, it is easy to detect a lesion.

There has also been proposed an endoscope system that applies excitationlight of a narrow band to a tissue inside the body cavity and performsdiagnosis using a fluorescence image created by acquiringautofluorescence generated from the tissue by the excitation light orchemical fluorescence (e.g., PTL 2).

When diagnosis is performed using an endoscope system that acquires afluorescence image, only a lesion such as a tumor emits fluorescence byusing a fluorescence agent that has a property of specificallyaccumulating on a lesion such as a tumor, so that the lesion can beeasily detected.

However, a narrow band light image and a fluorescence image(collectively referred to as special light images) typically have acolor that is largely different from the color of a normal light image.Further, these special light images are extremely dark due to the lackof illumination light. Thus, it is difficult to perform diagnosis byusing only a special light image. In view of this, in order to improvethe diagnostic accuracy for a user, for example, a normal light imageand a special light image may be simultaneously acquired and displayed.However, simultaneously displaying these images side by side causes auser to perform diagnosis while paying attention to a plurality ofimages all the time. This increases the load on the user. Further, auser may overlook a lesion by temporarily paying attention to a singleimage.

Therefore, PTL 3 has proposed that a first image corresponding to awavelength range of white light and a second image corresponding to aspecific wavelength range are acquired to determine the kind of asubject image within the second image and apply highlighting processingto the first image in accordance with the kind of the subject image tothereby prevent a lesion from being overlooked while reducing the loadon a user.

CITATION LIST Patent Literature PTL 1: JP 2006-68113 A PTL 2: JP2007-229053 A PTL 3: JP 2011-135983 A SUMMARY OF INVENTION TechnicalProblem

Methods in related art including PTLs 1 to 3 perform feature detectionbased on a feature amount obtained from a planar image by a 2D image.Thus, it is difficult to obtain information of the three-dimensional(3D) structure of a subject and the 3D positional relationship betweenobjects, for example, blood vessels. Therefore, it may be difficult toperform kind determination and feature detection on a subject image.

The present technology has been made in view of such circumstances toenable information of a 3D structure and the positional relationshipbetween objects to be easily acquired by using depth information.

Solution to Problem

A medical system according to an embodiment of the present technologyincludes: a medical imaging device, and an image processing apparatusfor processing an image captured by the medical imaging device thatincludes circuitry configured to acquire a special light image from theimage captured by the medical imaging device, the special light imagehaving information limited to a specific wavelength band, generate depthinformation at a predetermined position of a patient using the speciallight image, and detect a structural relationship using the depthinformation.

An image processing method according to an embodiment of the presenttechnology includes: acquiring a special light image, from the imagecaptured by the medical imaging device, the special light image havinginformation limited to a specific wavelength band, generating depthinformation at a predetermined position of a patient using the speciallight image, and detecting a structural relationship using the depthinformation.

A non-transitory computer readable medium having stored thereon aprogram that when executed by a computer causes the computer to executeprocessing. The processing according to an embodiment of the presenttechnology includes: acquiring a special light image, from the imagecaptured by the medical imaging device, the special light image havinginformation limited to a specific wavelength band, generating depthinformation at a predetermined position of a patient using the speciallight image, and detecting a structural relationship using the depthinformation.

An image processing apparatus for processing an image captured by themedical imaging device according to an embodiment of the presenttechnology includes: circuitry configured to acquire a special lightimage from the image captured by the medical imaging device, the speciallight image having information limited to a specific wavelength band,generate depth information at a predetermined position of a patientusing the special light image, and detect a structural relationshipusing the depth information.

Advantageous Effect of Invention

An embodiment of the present technology enables information of a 3Dstructure and the positional relationship between objects to be easilyacquired by using depth information.

The effects of the present technology are not necessarily limited to theeffect described herein and may be any of the effects described in thepresent disclosure.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating a configuration of an embodiment of animage processing apparatus to which the present technology is applied.

FIG. 2 is a diagram illustrating another configuration of the embodimentof the image processing apparatus to which the present technology isapplied.

FIG. 3 is a diagram illustrating still another configuration of theembodiment of the image processing apparatus to which the presenttechnology is applied.

FIG. 4 is a flowchart for describing processing of the image processingapparatus.

FIG. 5 is a diagram for describing a principle of detecting a bleedingposition.

FIG. 6 is a flowchart for describing processing of the image processingapparatus when a bleeding position is detected.

FIG. 7 is a diagram for describing a principle of detecting atransparent membrane.

FIG. 8 is a flowchart for describing processing of the image processingapparatus when a transparent membrane is detected.

FIG. 9 is a diagram for describing a principle of detecting a mist.

FIG. 10 is a flowchart for describing processing of the image processingapparatus when a mist is detected.

FIGS. 11A to 11C are diagrams for describing the positional relationshipbetween an artery and a vein in a 2D image.

FIG. 12 is a diagram for describing the reflectance of hemoglobin.

FIG. 13 is a diagram for describing a principle of detecting an arteryand a vein.

FIG. 14 is a flowchart for describing processing of the image processingapparatus when an artery and a vein are detected.

FIG. 15 is a diagram for describing determination of overlapping ofblood vessels.

FIG. 16 is an image example taking the positional relationship betweenblood vessels into consideration.

FIG. 17 is a flowchart for describing processing of the image processingapparatus when overlapping of blood vessels is detected.

FIG. 18 is a diagram for describing a principle of detecting the volumeof a tumor.

FIG. 19 is a flowchart for describing processing of the image processingapparatus when the volume of a tumor is detected.

FIG. 20 is a diagram for describing a recording medium.

DESCRIPTION OF EMBODIMENTS

Hereinbelow, a mode for carrying out the present technology(hereinbelow, referred to as an embodiment) is described. Thedescription is made in the following order.

1. Configuration of Image Processing Apparatus

2. Operation of Image Processing Apparatus

3. Application to Detection of Bleeding Position

4. Application to Detection of Transparent Membrane Thickness

5. Application to Detection of Mist

6. Application to Detection of Artery and Vein

7. Application to Detection of Overlapping State of Blood Vessels

8. Application to Detection of Tumor

9. Recording Medium

<Configuration of Image Processing Apparatus>

FIG. 1 is a diagram illustrating a configuration of an embodiment of animage processing system that includes an image processing apparatus towhich the present technology is applied. An image processing apparatus101 to which the present technology is applied is applicable to anapparatus that constitutes a part of an endoscope system or a microscopesystem and applicable to an apparatus that processes an image obtainedby imaging the inside of the body cavity of a subject.

The image processing apparatus 101 processes an image captured by animaging unit 102 and outputs the processed image to a display unit 103.The display unit 103 includes a display and displays an image processedby the image processing apparatus 101.

FIG. 1 illustrates the image processing apparatus 101, the imaging unit102, and the display unit 103 as different bodies. Alternatively, theimage processing apparatus 101 may include a part of the imaging unit102 or the entire imaging unit 102, or may include a part of the displayunit 103 or the entire display unit 103.

The image processing apparatus 101 includes a normal light imageacquisition unit 121, a special light image acquisition unit 122, a 3Dimage acquisition unit 123, a depth information generation unit 124, atarget region detection unit 125, and an image generation unit 126.

The imaging unit 102 includes a normal light image capture unit 141, aspecial light image capture unit 142, and a 3D image capture unit 143.

The image processing system illustrated in FIG. 1 is applicable to anendoscope system as described above. When the image processing systemillustrated in FIG. 1 is applied to an endoscope system, the imagingunit 102 is formed in a shape insertable into the body cavity of asubject.

For example, the imaging unit 102 is formed in an elongated and bendableshape so as to be insertable into the body cavity. The imaging unit 102has a detachable structure so that different imaging units are useddepending on regions to be observed. In the endoscopic field, theimaging unit 102 is typically called a scope. Concrete examples of ascope to be used include an upper digestive organ scope and a lowerdigestive organ scope.

Although not illustrated, the imaging unit 102 includes a light sourceunit for the purpose of imaging a dark part. Normal light is used as alight source, and the normal light image capture unit 141 captures animage of a part irradiated with the normal light. Special light is usedas a light source, and the special light image capture unit 142 capturesan image of a part irradiated with the special light. Alternatively, thespecial light image capture unit 142 applies normal light and capturesan image of light that is emitted from a part irradiated with the normallight and passes through a color filter having a specific color.

The 3D image capture unit 143 captures an image of a part irradiatedwith at least either normal light or special light. The captured imageis a stereoscopic image (3D image).

As illustrated in FIG. 2 and described below, a special light image maybe generated from an image captured by the normal light image captureunit 141 without providing the special light image capture unit 142illustrated in FIG. 1. Infrared light (IR) or narrow band wavelengthlight is used as the special light.

FIG. 1 illustrates, as an example, the 3D image capture unit 143 whichis provided separately from the normal light image capture unit 141 andthe special light image capture unit 142. Alternatively, as describedbelow with reference to FIG. 3, the normal light image capture unit 141and the special light image capture unit 142 may capture 3D images,respectively.

The normal light image acquisition unit 121 of the image processingapparatus 101 acquires image data of an image captured by the normallight image capture unit 141. The normal light image acquisition unit121 is provided with an analog-digital converter (AD converter, notillustrated). The normal light image acquisition unit 121 may convert ananalog image signal to a digital image signal to acquire image data ormay acquire digital image data from the normal light image capture unit141.

Similarly, the special light image acquisition unit 122 of the imageprocessing apparatus 101 acquires image data of an image captured by thespecial light image capture unit 142. In the following description, animage acquired by the special light image acquisition unit 122 isreferred to as a special light image. An image acquired by the normallight image acquisition unit 121 is referred to as a normal light image.An image acquired by the 3D image acquisition unit 123 is referred to asa 3D image.

The depth information generation unit 124 of the image processingapparatus 101 generates depth information from a 3D image acquired bythe 3D image acquisition unit 123. The target region detection unit 125detects a predetermined region using depth information supplied from thedepth information generation unit 124 and a special light image suppliedfrom the special light image acquisition unit 122.

For example, the target region detection unit 125 generates informationsuch as the thickness of a predetermined membrane and the positionalrelationship between blood vessels, for example, between a front bloodvessel and a rear blood vessel. Concrete examples of how to use thedepth information and information to be generated are described below.

The image generation unit 126 generates an image to be provided to auser using a normal light image supplied from the normal light imageacquisition unit 121, a special light image supplied from the speciallight image acquisition unit 122, a 3D image supplied from the 3D imageacquisition unit 123, and information about a target region detected bythe target region detection unit 125.

For example, an image in which information about a target region, forexample, a numerical value of the membrane thickness is superimposed ona normal light image is generated. Displaying visually recognizableinformation, for example, a numerical value of the membrane thickness onan image, for example, a normal light image, a special light image, or a3D light image in this manner enables a user to obtain information thatis difficult to obtain merely by looking at the normal light image, thespecial light image, or the 3D light image. Thus, the usability isobviously improved.

FIG. 2 is a diagram illustrating another configuration of the embodimentof the image processing system which includes the image processingapparatus to which the present technology is applied. A part of theimage processing system illustrated in FIG. 2, the part having aconfiguration similar to that of the image processing system illustratedin FIG. 1 is designated by the same reference sign. Hereinbelow,description of the similar part is appropriately omitted, and adifferent part is described.

The image processing system illustrated in FIG. 2 also includes an imageprocessing apparatus 201, an imaging unit 202, and a display unit 103.The image processing apparatus 201 has the same configuration as theimage processing apparatus 101 illustrated in FIG. 1 excepting that aspecial light image acquisition unit 222 acquires a normal light imagefrom a normal light image capture unit 141 of the imaging unit 202.

The imaging unit 202 includes the normal light image capture unit 141and a 3D image capture unit 143. The imaging unit 202 illustrated inFIG. 2 differs from the imaging unit 102 illustrated in FIG. 1 in thatno special light image capture unit 142 is provided.

The special light image acquisition unit 222 of the image processingapparatus 201 receives the supply of a normal light image from thenormal light image capture unit 141. Although, here, the special lightimage acquisition unit 222 is described to receive the supply of anormal light image from the normal light image capture unit 141, thespecial light image acquisition unit 222 may receive the supply of anormal light image acquired by the normal light image acquisition unit121 from the normal light image capture unit 141.

The special light image acquisition unit 222 has a function ofgenerating a special light image from a normal light image. The speciallight image is an image captured with light in a predetermined band, forexample, an image obtained by imaging a part that reacts to blue lightwhen irradiated with the blue light. The special light image acquisitionunit 222 extracts a blue component image from a normal light image togenerate a special light image.

The image processing system illustrated in FIG. 2 acquires a speciallight image from a normal light image in this manner.

FIG. 3 is a diagram illustrating still another configuration of theembodiment of the image processing system which includes the imageprocessing apparatus to which the present technology is applied. A partof the image processing system illustrated in FIG. 3, the part having aconfiguration similar to that of the image processing system illustratedin FIG. 1 is designated by the same reference sign. Hereinbelow,description of the similar part is appropriately omitted, and adifferent part is described.

An imaging unit 302 of the image processing system illustrated in FIG. 3includes a normal light 3D image capture unit 341 and a special light 3Dimage capture unit 342. The normal light 3D image capture unit 341captures a normal light 3D image obtained under normal light. Thespecial light 3D image capture unit 342 captures a special light 3Dimage obtained under special light.

A normal light image acquisition unit 321 of an image processingapparatus 301 acquires a normal light 3D image captured by the normallight 3D image capture unit 341. The normal light image acquisition unit321 may generate a 2D image from the acquired normal light 3D image andsupply the generated normal light 2D image to an image generation unit126, or may supply the acquired normal light 3D image to the imagegeneration unit 126 to provide a user with the normal light 3D image asit is.

A special light image acquisition unit 322 of the image processingapparatus 301 acquires a special light 3D image captured by the speciallight 3D image capture unit 342. The special light image acquisitionunit 322 may generate a 2D image from the acquired special light 3Dimage and supply the generated special light 2D image to the imagegeneration unit 126, or may supply the acquired special light 3D imageto the image generation unit 126 to provide a user with the speciallight 3D image as it is.

A 3D image acquisition unit 323 acquires a normal light 3D image fromthe normal light 3D image capture unit 341 and a special light 3D imagefrom the special light 3D image capture unit 342. A depth informationgeneration unit 124 generates depth information from the acquired normallight 3D image and the acquired special light 3D image depending on whatis to be detected by a target region detection unit 125.

In each of the image processing apparatuses illustrated in FIGS. 1 to 3,a user may select which one is to be displayed on the display unit 103among the normal light image, the special light image, and the 3D image.Although not illustrated, the image processing apparatus is alsoprovided with an operation unit which receives such an instruction froma user.

The configurations of the image processing apparatuses described hereinare merely examples, and indicate no limitation. For example, in theimage processing apparatus 301 illustrated in FIG. 3, a special lightimage may be generated from a normal light image as with the imageprocessing apparatus 201 illustrated in FIG. 2. In such a configuration,a normal light 3D image may be supplied from the normal light 3D imagecapture unit 341 of the imaging unit 302 to the special light imageacquisition unit 322 of the image processing apparatus 301.

That is, as described above, a special light image having information ina specific wavelength band may be acquired by performing imaging usinglight in the specific wavelength band or may be acquired by extractinginformation in the specific wavelength band from a normal light imagecaptured using light in a wavelength band of normal light (white light).

Further, a 3D image may be acquired by performing stereoscopic imagecapturing or may be acquired by acquiring a 2D image and then convertingthe 2D image to the 3D image.

Next, processing executed in the image processing apparatusesillustrated in FIGS. 1 to 3 is described. First, the processing of theimage processing apparatuses illustrated in FIGS. 1 to 3 is described inoutline with reference to a flowchart of FIG. 4. Then, description withconcrete examples of a region to be detected is made.

<Operation of Image Processing Apparatus>

The processing of the image processing apparatus is described in outlinewith reference to the flowchart of FIG. 4. Although, here, the imageprocessing apparatus 301 illustrated in FIG. 3 is described as anexample, processing is basically performed in a similar manner also inthe image processing apparatus 101 illustrated in FIG. 1 and the imageprocessing apparatus 201 illustrated in FIG. 2.

In step S101, the 3D image acquisition unit 323 acquires a normal light3D image from the normal light 3D image capture unit 341 and acquires aspecial light 3D image from the special light 3D image capture unit 342.In step S102, the depth information generation unit 124 generates depthinformation from the 3D images acquired by the 3D image acquisition unit323. The depth information is generated using coordinates of the 3Dimages at a predetermined position, in particular, coordinates in thedepth direction.

In step S103, the target region detection unit 125 detects apredetermined region, for example, a blood vessel or a tumor using thedepth information and a special light image acquired by the speciallight image acquisition unit 322. In step S104, the image generationunit 126 generates an image that clearly shows the detected region to auser and outputs the generated image to the display unit 103.

In this manner, the image processing apparatus to which the presenttechnology is applied is capable of performing more detailed detectionof a target region in more detail by also using depth informationobtained from 3D images. Further, it is also possible to present thedepth information to a user. Thus, for example, positional informationof a blood vessel can be presented in more detail.

<Application to Detection of Bleeding Position>

Next, the processing of the image processing apparatus 301 is furtherdescribed with a concrete example in which a bleeding region is detectedas a region to be detected. First, a principle of detecting a bleedingposition is described with reference to FIG. 5.

In FIG. 5, a flesh surface 501 is illustrated on the lower side. Apuddle of blood is formed on the flesh surface 501 by bleeding. Thebuddle of blood has a blood surface 502. It is conceivable that ableeding part of the puddle of blood rises higher than the other part ofthe puddle of blood. It is conceivable that, if the flesh surface 501 isflat, as illustrated in FIG. 5, a bleeding point P1 rises higher thanthe other part.

In practice, the flesh surface 501 may not be flat, but may be uneven.However, the difference between the flesh surface 501 and the bloodsurface 502, that is, the thickness of the puddle of blood is thick atthe bleeding point P1. Thus, a 3D image of the flesh surface 501 and a3D image of the blood surface 502 are acquired. Then, the differencebetween the 3D image of the flesh surface 501 and the 3D image of theblood surface 502 is calculated to measure the thickness of the puddleof blood at each position. When the thickness at a position is equal toor larger than a certain thickness, the position can be detected to be ableeding point.

When infrared light (IR) is used as light during image capturing inimaging of the flesh surface 501 and the blood surface 502, the infraredlight passes through the blood surface 502 and reaches the flesh surface501. A 3D image of the flesh surface 501 is acquired by performingstereoscopic image capturing using infrared light by using such acharacteristic.

On the other hand, when white light is used as light during imagecapturing, the white light is reflected by the blood surface 502 toreturn without passing through the blood surface 502. A 3D image of theblood surface 502 is acquired by performing stereoscopic image capturingusing white light by using such a characteristic.

The normal light 3D image capture unit 341 of the imaging unit 302 (FIG.3) performs image capturing using white light to acquire a 3D image ofthe blood surface 502. The special light 3D image capture unit 342performs image capturing using infrared light to acquire a 3D image ofthe flesh surface 501. The difference between the two images iscalculated to obtain thickness information of the puddle of blood ateach point.

Referring to FIG. 5, at a point P2, a depth to the flesh surface 501imaged using infrared light (IR) is denoted by depth IR_depth and adepth to the blood surface 502 imaged using white light (WL) is denotedby depth WL_depth. In this case, thickness s_depth of the puddle ofblood at the point P2 can be represented by the following Equation 1.

thickness s_depth=depth IR_depth−depth WL_depth

When the depth WL_depth obtained by such an arithmetic operation isgreater than a predetermined threshold, the thickness of blood is large.Thus, such a position can be detected as a bleeding position. In thismanner, a position having a depth WL_depth greater than the threshold isdetermined to be a bleeding position. Thus, it is possible to cope witha case in which there is a plurality of bleeding positions.

For example, when a position having the largest measured value isdetermined to be a bleeding position, only one position can be detected.On the other hand, since a position having a measured value equal to orgreater than the threshold is determined to be a bleeding position, evenwhen there is a plurality of bleeding positions, each of the positionscan be determined to be a bleeding position.

An operation of the image processing apparatus 301 (FIG. 3) whichperforms such processing is described with reference to a flowchart ofFIG. 6.

In step S201, a normal light 3D image is acquired. In this case, thenormal light 3D image capture unit 341 of the imaging unit 302 performsimaging with white light to acquire the normal light 3D image, and theacquired normal light 3D image is supplied to the 3D image acquisitionunit 323.

In step S202, a special light 3D image is acquired. In this case, thespecial light 3D image capture unit 342 of the imaging unit 302 performsimaging with infrared light to acquire the special light 3D image, andthe acquired special light 3D image is supplied to the 3D imageacquisition unit 323.

In step S203, the depth information generation unit 124 calculates adepth difference. The depth difference is calculated by performing thearithmetic operation of the above Equation 1 using coordinates in thedepth direction of the 3D images. The arithmetic operation may beexecuted for all points (pixels) within the acquired image or an areathat is determined to have a puddle of blood to generate thicknessinformation of blood, or the acquired image may be divided into areaseach having a predetermined size and the arithmetic operation may beexecuted for each of the areas to generate thickness information ofblood.

In step S204, it is determined whether the differential value calculatedin step S203 is equal to or greater than the threshold. Thedetermination may be made by the depth information generation unit 124and the determined result may be supplied to the target region detectionunit 125. Alternatively, the target region detection unit 125 mayreceive the supply of depth information (differential value) generatedby the depth information generation unit 124 to make the determination.

When the differential value is determined to be equal to or greater thanthe threshold in step S204, the processing proceeds to step S205. Instep S205, a current processing point at this point of time is set as ableeding point. The point set as the bleeding point is displayed in amanner to be uniquely distinguishable from the other part of the puddleof blood within an image to be provided to a user, for example, thebleeding point is displayed with a predetermined mark, or text thatindicates being the bleeding point is displayed. Such display may besuperimposed on a normal light image or a special light image. Further,the image to be superimposed may be a 3D image.

On the other hand, when the differential value is determined to be lessthan the threshold in step S204, the processing proceeds to step S206.In step S206, a current processing point at this point of time is set asa part of the puddle of blood other than the bleeding point.

In this manner, it is possible to obtain a differential value from two3D images and detect a bleeding point from the obtained differentialvalue. Information about the detected bleeding point is presented to auser. Thus, a user can recognize the bleeding point, which is difficultto recognize merely by looking at the captured image, by browsing theinformation.

<Application to Detection of Transparent Membrane Thickness>

Next, the processing of the image processing apparatus 301 is furtherdescribed with a concrete example in which a transparent membrane isdetected as a region to be detected. First, a principle of detecting atransparent membrane is described with reference to FIG. 7.

In FIG. 7, a flesh surface 601 is illustrated on the lower side. Atransparent membrane 603 having a predetermined thickness is located onthe flesh surface 601. The transparent membrane 603 has a transparentmembrane surface 602. It is difficult to capture an image of atransparent membrane with normal light (white light). Thus, it isdifficult for a user to determine whether the transparent membrane ispresent in an image captured with white light.

However, it may be necessary to cut a transparent membrane to get theflesh surface 601. In such a case, presenting the thickness of thetransparent membrane to a user makes it easy to know to what extent thetransparent membrane should be cut. Accordingly, it is possible toimprove the usability of apparatuses such as endoscopes and microscopesto which the image processing apparatus 301 is applied.

An image of the transparent membrane surface 602 can be captured bypolarization image capturing. As illustrated in FIG. 7, polarized lightP is reflected by the transparent membrane surface 602. Thus, thetransparent membrane surface 602 can be imaged by imaging the reflectedlight. The flesh surface 601 can be imaged by imaging using white light(WL).

The normal light 3D image capture unit 341 of the imaging unit 302 (FIG.3) performs stereoscopic image capturing using white light to acquire a3D image of the flesh surface 601. The special light 3D image captureunit 342 performs stereoscopic image capturing using polarized light toacquire a 3D image of the transparent membrane surface 602. Thedifference between the two images is calculated to obtain thicknessinformation of the transparent membrane 603 at each point.

Referring to FIG. 7, at a point P1, a depth to the flesh surface 601imaged using white light (WL) is denoted by depth WL_depth and a depthto the transparent membrane surface 602 imaged using polarized light (P)is denoted by depth P_depth. In this case, membrane thickness f_depth ofthe transparent membrane 603 at the point P1 can be represented by thefollowing Equation 2.

membrane thickness f_depth=depth WL_depth−depth P_depth

The membrane thickness f_depth obtained by such an arithmetic operationis defined as the thickness of the transparent membrane. An operation ofthe image processing apparatus 301 (FIG. 3) which performs suchprocessing is described with reference to a flowchart of FIG. 8.

In step S301, a normal light 3D image is acquired. In this case, thenormal light 3D image capture unit 341 of the imaging unit 302 performsstereoscopic imaging with white light to acquire the normal light 3Dimage, and the acquired normal light 3D image is supplied to the 3Dimage acquisition unit 323.

In step S302, a special light 3D image is acquired. In this case, thespecial light 3D image capture unit 342 of the imaging unit 302 performspolarization stereoscopic imaging to acquire the special light 3D image,and the acquired special light 3D image is supplied to the 3D imageacquisition unit 323.

In step S303, the depth information generation unit 124 calculates adepth difference. The depth difference is calculated by performing thearithmetic operation of the above Equation 2 using coordinates in thedepth direction of the 3D images. The differential value calculated instep S303 is set as the membrane thickness of the transparent membrane603.

In this manner, the membrane thickness of the transparent membrane isdetected. As the result of the arithmetic operation of Equation 2, whenthe differential value is zero or equal to or less than a predeterminedthreshold, it can be determined that there is no transparent membrane.That is, determining whether the differential value is equal to orgreater than a predetermined threshold also enables the presence/absenceof the transparent membrane 603 to be detected.

The membrane thickness of the transparent membrane detected in thismanner is displayed as a numerical value or color information such asgradation within an image to be provided to a user. The display enablesa user to recognize the presence/absence of the transparent membrane andthe thickness of the transparent membrane at a glance. Such display maybe superimposed on a normal light image or a special light image.Further, the image to be superimposed may be a 3D image.

For example, a numerical value of the membrane thickness may bedisplayed at a preset position, for example, a central part of a screenor the position of lattice points at a certain interval. Alternatively,display may be performed in such a manner that the color is changedcorresponding to the membrane thickness and superimposed on a normallight image. Alternatively, a numerical value of the membrane thicknessmay be displayed at a position instructed by a user by operating anoperation unit such as a pointer and a mouse.

In this manner, it is possible to obtain a differential value from two3D images and detect a transparent membrane and the membrane thicknessof the transparent membrane from the obtained differential value.Information about the detected transparent membrane is presented to auser. Thus, a user can recognize the transparent membrane, which isdifficult to recognize merely by looking at the captured normal lightimage, by browsing the information. Since the membrane thickness of thetransparent membrane can be recognized, a sense of putting a scalpel iseasily got. This prevents flesh tissues from being erroneously damaged.

<Application to Detection of Mist>

Next, the processing of the image processing apparatus 301 is furtherdescribed with a concrete example in which a mist is detected as aregion to be detected. First, a principle of detecting a mist isdescribed with reference to FIG. 9.

In FIG. 9, a flesh surface 701 is illustrated on the lower side. A mist703 having a predetermined thickness is located on the flesh surface701. The mist 703 has a mist surface 702. When imaging is performed withnormal light (white light), a mist component is reflected and imaged asa white point. A shine or glare caused by the structure of the fleshsurface 701 may also be recognized as a white point. It is difficult todistinguish the mist and the shine in an image captured with normallight.

An image of the flesh surface 701 can be captured by infrared lightimage capturing. As illustrated in FIG. 9, infrared light IR passesthrough the mist surface 702 and is then reflected by the flesh surface701. Thus, the flesh surface 701 can be imaged by imaging the reflectedlight. The mist surface 702 can be imaged by imaging using white light(WL).

The normal light 3D image capture unit 341 of the imaging unit 302 (FIG.3) performs stereoscopic image capturing using white light to acquire a3D image of the mist surface 702. The special light 3D image captureunit 342 performs stereoscopic image capturing using infrared light toacquire a 3D image of the flesh surface 701. The difference between thetwo images is calculated to obtain thickness information of the mist 703at each point.

Referring to FIG. 9, at a point P1, a depth to the mist surface 702imaged using white light (WL) is denoted by depth WL_depth and a depthto the flesh surface 701 imaged using infrared light (IR) is denoted bydepth IR_depth. In this case, thickness m_depth of the mist 703 at thepoint P1 can be represented by the following Equation 3.

thickness m_depth=depth IR_depth−depth WL_depth

When the thickness m_depth obtained by such an arithmetic operation isgreater than a predetermined threshold th, a mist is determined to bepresent. An operation of the image processing apparatus 301 (FIG. 3)which performs such processing is described with reference to aflowchart of FIG. 10.

In step S401, a normal light 3D image is acquired. In this case, thenormal light 3D image capture unit 341 of the imaging unit 302 performsimaging with white light to acquire the normal light 3D image, and theacquired normal light 3D image is supplied to the 3D image acquisitionunit 323.

In step S402, a special light 3D image is acquired. In this case, thespecial light 3D image capture unit 342 of the imaging unit 302 performsimaging with infrared light to acquire the special light 3D image, andthe acquired special light 3D image is supplied to the 3D imageacquisition unit 323.

In step S403, the depth information generation unit 124 calculates adepth difference. The depth difference is calculated by performing thearithmetic operation of the above Equation 3 using coordinates in thedepth direction of the 3D images.

In step S404, it is determined whether the differential value calculatedin step S403 is equal to or greater than the threshold. Thedetermination may be made by the depth information generation unit 124and the determined result may be supplied to the target region detectionunit 125. Alternatively, the target region detection unit 125 mayreceive the supply of depth information (differential value) generatedby the depth information generation unit 124 to make the determination.

When the differential value is determined to be equal to or greater thanthe threshold in step S404, the processing proceeds to step S405. Instep S405, a current processing point at this point of time is set as amist. The threshold is a value obtained by adding a predetermined valueto a value taking unevenness of the flesh surface 701 intoconsideration.

On the other hand, when the differential value is determined to be lessthan the threshold in step S404, the processing proceeds to step S406.In step S406, a current processing point at this point of time is set asnot being a mist.

When a mist is detected, alarm display for notifying a user of thepresence of the mist or display in which a color representing the mistis superimposed on a part in which the mist has been detected isperformed. Such display may be superimposed on a normal light image or aspecial light image. Further, the image to be superimposed may be a 3Dimage.

When the ratio of a region in which the mist has been detected to theentire image is large, an image of the flesh surface 701 with no mistmay be provided to a user by black-and-white display using an IR lightsource (infrared light). Alternatively, image processing for removingthe mist may be applied to an image to provide the image from which themist has been removed to a user. This makes it possible to ensure anoperative field to prevent an operation error.

In this manner, it is possible to obtain a differential value from two3D images and detect a mist or a shine from the obtained differentialvalue. Information about the detected mist is presented to a user. Thus,a user can recognize the presence of a mist, which is difficult torecognize merely by looking at the captured image, by browsing theinformation.

<Application to Detection of Artery and Vein>

Next, the processing of the image processing apparatus 301 is furtherdescribed with a concrete example in which an artery or a vein isdetected as a region to be detected. First, an image obtained by imagingan artery and a vein with a 2D normal light is described with referenceto FIGS. 11A to 11C. Although, here, an artery and a vein are describedas an example, the present embodiment is not limited to a combination ofan artery and a vein. The present embodiment is applicable to bloodvessels.

FIG. 11A illustrates a state in which a vein 802 and an artery 803 arelocated at the same depth in the depth direction under a surface 801. InFIGS. 11A to 11C, the vein is illustrated as a shaded circle and theartery is illustrated as a white circle. FIG. 11B illustrates a state inwhich the vein 802 and the artery 803 are located at different depths inthe depth direction under the surface 801.

When the state illustrated in FIG. 11A and the state illustrated in FIG.11B are imaged as a 2D image with normal light from the surface 801, animage 811 as illustrated in FIG. 11C is captured. In the image 811, avein 812 and an artery 813 are imaged. Since the image 811 is a 2D imageand information indicating the positional relationship between the vein812 and the artery 813 is not displayed on the image 811, it isdifficult to read which one is located at a deeper position between thevein 812 and the artery 813.

FIG. 12 is a graph illustrating the reflectance in hemoglobin combinedwith oxygen (HbO₂) and the reflectance in hemoglobin separated fromoxygen (Hb). In the graph of FIG. 12, the horizontal axis represents thewavelength of light to be applied and the vertical axis represents thereflectance.

The graph of FIG. 12 shows that hemoglobin combined with oxygen (HbO₂)and hemoglobin separated from oxygen (Hb) have different reflectancecharacteristics by the wavelength. Hemoglobin combined with oxygen(HbO₂) flows through an artery and hemoglobin separated from oxygen (Hb)flows through a vein.

When an artery and a vein are located at the same depth and havesubstantially the same thickness, the artery has a higher reflectancethan the vein near a wavelength of 640 nm or larger to be more vivid reddue to a difference in reflectance, and is thus distinguishable from thevein. However, when the vein is located near the surface and the vein islocated at a slightly deep position, and the two blood vessels havesubstantially the same thickness as illustrated in FIG. 11B, lightreflected by flesh tissues before reaching the vein increases to causethe vein to have the same degree of brightness as the artery whenobserved from the surface with normal light. Thus, it is difficult todistinguish between two blood vessels, specifically, a vein and anartery located at different depths.

Distinction between the vein 812 and the artery 813 may be made from thepositional relationship in the depth direction therebetween. Thus, thepositional relationship in the depth direction between blood vessels isimportant information. Therefore, as illustrated in FIG. 13, bloodvessels located at different depths are imaged with differentwavelengths.

First, 3D images are captured with two beams of light having differentwavelengths. Here, the imaging is performed with a first wavelength λ1and a second wavelength λ2. The beams of light having differentwavelengths reach different depths within a flesh tissue. The firstwavelength λ1 reaches a relatively shallow part. The second wavelengthλ2 reaches a relatively deep part.

As illustrated in FIG. 13, when imaging is performed with the vein 802located near the surface 801 and the artery 803 located at a deepposition, the first wavelength λ1 is reflected by the vein 802, and thesecond wavelength λ2 is reflected by the artery 803. That is, depthλ1_depth as depth information of the vein 802 is acquired bystereoscopic image capturing with the first wavelength λ1, and depthλ2_depth as depth information of the artery 803 is acquired bystereoscopic image capturing with the second wavelength λ2.

The distance between the two blood vessels can be obtained bycalculating the difference between the two depths.

distance dif_depth=depth λ2_depth−depth λ1_depth

The distance information between the blood vessels obtained in thismanner is used to correct and display the reflectance of the arterylocated at a deep position. Accordingly, an image that enablesdistinction between the artery and the vein can be obtained.

An operation of the image processing apparatus 301 (FIG. 3) whichperforms such processing is described with reference to a flowchart ofFIG. 14.

In step S501, a 3D image captured with the first wavelength λ1 isacquired. In this case, the special light 3D image capture unit 342 ofthe imaging unit 302 performs imaging with the first wavelength λ1 toacquire the special light 3D image, and the acquired special light 3Dimage is supplied to the 3D image acquisition unit 323.

In step S502, a 3D image captured with the second wavelength λ2 isacquired. In this case, the special light 3D image capture unit 342 ofthe imaging unit 302 performs imaging with the second wavelength λ2 toacquire the special light 3D image, and the acquired special light 3Dimage is supplied to the 3D image acquisition unit 323.

In step S503, the depth information generation unit 124 calculates adepth difference.

The depth difference is calculated by performing the arithmeticoperation of the above Equation 4 using coordinates in the depthdirection of the 3D images. The calculated differential value is definedas the distance between the blood vessels.

An image that enables distinction between the artery and the vein can beobtained by correcting and displaying the reflectance of the arterylocated at a deep position using the distance between the blood vesselscalculated in this manner. For example, a blood vessel located at ashallow position is displayed in red, and a blood vessel that can bedetermined to be located at a deeper position than the shallow bloodvessel from distance information is displayed in slightly dark red toperform coloring corresponding to the depth. Accordingly, an image thatenables a user to recognize the positional relationship between bloodvessels corresponding to the depth at a glance is presented.

Such an image may be superimposed on a normal light image or a speciallight image to be presented to a user. Alternatively, the image may besuperimposed on a 3D image.

In this manner, it is possible to obtain a differential value from two3D images and detect the positional relationship (depth information)between blood vessels from the obtained differential value. The detectedpositional information is presented to a user. Thus, a user canrecognize the positional relationship between the blood vessels, whichis difficult to recognize merely by looking at the captured image, bybrowsing the information.

<Application to Detection of Overlapping State of Blood Vessels>

Next, the processing of the image processing apparatus 301 is furtherdescribed with a concrete example in which an overlapping state of bloodvessels is detected as a region to be detected. First, an overlappingstate of blood vessels is described with reference to FIG. 15.

For example, the deep part of a mucous membrane or placental bloodvessels in turbid amniotic fluid can be observed by excitation lightobservation after indocyanine green (ICG) injection. However, in a parthaving a plurality of overlapping blood vessels as illustrated in theleft figure in FIG. 15, it is difficult to determine an overlappingstate of the blood vessels. Thus, the exit of a blood vessel that isconnected to a cancer tissue 911 and thus has metastatic potential maybe erroneously determined to be the exit of another blood vessel.

In the left figure in FIG. 15, a blood vessel 901, a blood vessel 902,and a blood vessel 903 overlap each other on the central part in thefigure. Thus, it is difficult to determine whether the blood vessels areconnected as illustrated in the upper right figure in FIG. 15 or theblood vessels are connected as illustrated in the lower right figure inFIG. 15.

The upper right figure in FIG. 15 illustrates a state when it isdetermined that the blood vessel 901 and a blood vessel 904 areconnected, the blood vessel 902 and a blood vessel 905 are connected,and the blood vessel 903 and a blood vessel 906 are connected. Based onsuch determination, the blood vessel 903 and the blood vessel 906 areconnected to the cancer tissue 911.

The lower right figure in FIG. 15 illustrates a state when it isdetermined that the blood vessel 901 and the blood vessel 904 areconnected, the blood vessel 902 and the blood vessel 906 are connected,and the blood vessel 903 and the blood vessel 905 are connected. Basedon such determination, the blood vessel 903 and the blood vessel 905 areconnected to the cancer tissue 911.

In this manner, when an overlapping state is not correctly determined,connection of blood vessels may be erroneously determined. Thus, it isimportant to present an overlapping state between blood vessels to auser to enable the user to determine connection of the blood vessels.

Thus, stereoscopic image capturing is performed under irradiation withexcitation light after ICG injection to detect distance informationICG_depth. Then, connection and overlapping of blood vessels (that is,distinction between different blood vessels) are detected from thedetected distance information ICG_depth. Further, the direction of ablood flow is determined from the blood vessel overlapping informationto estimate a region having a high metastatic potential of a cancertissue.

For example, an image as illustrated in FIG. 16 is presented to a user.The image example illustrated in FIG. 16 displays that the blood vessel901 and the blood vessel 904 are connected, the blood vessel 902 and theblood vessel 905 are connected, and the blood vessel 903 and the bloodvessel 906 are connected. The connection of the blood vessels aredetected in such a manner that stereoscopic image capturing is performedunder irradiation with excitation light after ICG injection to detectdistance information ICG_depth, and the connection and overlapping ofthe blood vessels are detected from the detected distance informationICG_depth.

Further, the positional relationship between the blood vessels,specifically, the blood vessel 901 and the blood vessel 904 are locatedon the top, the blood vessel 902 and the blood vessel 905 are located onthe bottom, and the blood vessel 903 and the blood vessel 906 areinterposed therebetween, is also detected from the distance informationICG_depth. Thus, display that makes such a positional relationship clearis performed.

For example, in the display of overlapping blood vessels, differentblood vessels are displayed with different brightnesses or colors toenable a user to easily determine the blood vessel structure. A regionthat has metastatic potential of cancer may be estimated from connectioninformation of a blood vessel connected to the cancer tissue 911, andthe region may be highlighted.

Such display (presentation of an image) enables a user to correctlydetermine an overlapping sate of blood vessels. Thus, an infiltratedblood vessel/tissue can be correctly removed. Further, it is possible toreduce overlook of a region that has metastatic potential of cancer.

An operation of the image processing apparatus 301 (FIG. 3) whichperforms such processing is described with reference to a flowchart ofFIG. 17.

In step S601, a 3D image is acquired by excitation light imagecapturing. In this case, the special light 3D image capture unit 342 ofthe imaging unit 302 performs imaging with excitation light after ICGinjection to acquire the special light 3D image, and the acquiredspecial light 3D image is supplied to the 3D image acquisition unit 323.

In step S602, the depth information generation unit 124 calculates adepth difference.

In step S603, connection and overlapping of blood vessels are detected.The connection and the overlapping of blood vessels are detected in sucha manner that the 3D image acquired by the excitation light imagecapturing is analyzed to detect a vertical positional relationshipbetween the blood vessels from the depth information of each bloodvessel in a part in which the blood vessels overlap each other or toestimate the connection of blood vessels located at substantially thesame depth from the depth information.

As described above, different blood vessels can be displayed withdifferent brightnesses or colors or a region that has metastaticpotential of cancer can be highlighted using the detected result. Thus,a user can correctly determine an overlapping state of blood vessels. Asa result, an infiltrated blood vessel/tissue can be correctly removed.Further, it is possible to reduce overlook of a region that hasmetastatic potential of cancer.

<Application to Detection of Tumor>

Next, the processing of the image processing apparatus 301 is furtherdescribed with a concrete example in which a tumor is detected as aregion to be detected. First, estimation of the volume of a tumor isdescribed with reference to FIG. 18.

In photodynamic diagnosis (PDD), observation is performed with bluelight after a patient takes aminolevulinic acid (5-ALA), and a tumorlooks like emitting red light. In this PDD, only a planar image can beobserved. Thus, it is difficult for a user to recognize a real size of atumor.

For example, as illustrated in FIG. 18, a tumor may be a tumor 1001which has a planar expansion, but does not infiltrate deep in an innerface direction, or may be a tumor 1002 which has little planarexpansion, but infiltrates deep in the inner face direction. It isdifficult to determine whether a tumor infiltrates deep as with thetumor 1002 only with a planar image.

Thus, stereoscopic image capturing is performed under PDD to detectdistance information PDD_depth, and the volume of a tumor is estimatedfrom the surface area of the tumor obtained from a 2D image and thedistance information PDD_depth. A result of the volume estimationperformed in this manner is presented to a user by, for example,highlighting a tumor having a large estimated volume.

A tumor that has a small surface area on a 2D image and has a largeestimated volume may be highlighted in a further distinguishable manner.Such display enables a user to correctly distinguish a tumor having alarge volume which is likely to be brought to follow-up due to its smallsurface area and to determine an appropriate treatment.

An operation of the image processing apparatus 301 (FIG. 3) whichperforms such processing is described with reference to a flowchart ofFIG. 19.

In step S701, a 2D image is acquired by imaging under PDD. In this case,the special light 3D image capture unit 342 of the imaging unit 302performs stereoscopic imaging under irradiation with blue light after apatient takes aminolevulinic acid to acquire a special light 3D image inwhich an affected part is colored in red. The acquired 3D image isconverted into the 2D image to acquire the 2D image.

In step S702, a 3D image is acquired by imaging under PDD. As with stepS701, the special light 3D image capture unit 342 of the imaging unit302 performs stereoscopic imaging under irradiation with blue lightafter a patient takes aminolevulinic acid to acquire the special light3D image in which an affected part is colored in red.

In step S703, the depth information generation unit 124 calculates thedepth of a tumor (the depth in the inner face direction) using theacquired special light 3D image.

In step S704, the volume of the tumor is estimated. The surface area iscalculated from the acquired special light 2D image and multiplied bythe depth information calculated from the 3D image to estimate thevolume of the tumor.

In accordance with the volume estimated in this manner, as describedabove, a tumor having a large estimated volume is, for example,highlighted to be presented to a user. Such display enables a user tocorrectly determine the size of the tumor and determine an appropriatetreatment.

In this manner, the present technology makes it possible to detect atarget region and a feature amount related thereto, which are difficultto detect by detection processing only with a 2D image. Further, a usercan obtain more pieces of information in an easily visually recognizableform by performing highlighting processing or image superimposingprocessing based on the detected target region and feature amount on thepresented image.

This makes it possible to achieve more accurate diagnosis before, duringand after an operation, improvement in the accuracy of an operation,reduction in operation time, and minimally invasive operation.

The above detection operations of the specific regions may be performedindependently or in combination. For example, since the detection of anartery and a vein and the detection of an overlapping state of bloodvessels are both blood vessel detection operations, these detectionoperations may be performed in combination to more appropriately detectan artery and a vein and clearly detect the positional relationshipbetween the artery and the vein.

Further, the detection operations of different regions may be switchedto be performed. For example, a transparent membrane and a mist may besequentially detected and the presence of the transparent membrane andthe mist may be presented to a user, and detection of a tumor may alsobe performed.

<Recording Medium>

The above series of processing may be executed by hardware or software.When the series of processing is executed by software, a programconstituting the software is installed in a computer. The computerincludes a computer incorporated into dedicated hardware and, forexample, a general personal computer capable of executing variousfunctions by installing various programs therein.

FIG. 20 is a block diagram illustrating an example of the configurationof hardware of a computer which executes the above series of processingby a program. In the computer, a central processing unit (CPU) 2001, aread only memory (ROM) 2002, and a random access memory (RAM) 2003 areconnected to each other through a bus 2004. Further, an input/outputinterface 2005 is connected to the bus 2004. An input unit 2006, anoutput unit 2007, a storage unit 2008, a communication unit 2009, and adrive 2010 are connected to the input/output interface 2005.

The input unit 2006 includes a keyboard, a mouse, and a microphone. Theoutput unit 2007 includes a display and a speaker. The storage unit 2008includes a hard disk and a nonvolatile memory. The communication unit2009 includes a network interface. The drive 2010 drives a removablemedium 2011 which includes a magnetic disk, an optical disk, amagneto-optical disk, and a semiconductor memory.

In the computer having the above configuration, for example, the CPU2001 loads a program stored in the storage unit 2008 to the RAM 2003through the input/output interface 2005 and the bus 2004 to execute theprogram to perform the above series of processing.

For example, the program executed by the computer (CPU 2001) may beprovided by recording the program in the removable medium 2011 as apackage medium. Alternatively, the program may be provided through awired or wireless transmission medium such as a local area network(LAN), Internet, and digital satellite broadcasting.

In the computer, the program may be installed in the storage unit 2008through the input/output interface 2005 by attaching the removablemedium 2011 to the drive 2010. Alternatively, the program may bereceived by the communication unit 2009 through a wired or wirelesstransmission medium and installed in the storage unit 2008.Alternatively, the program may be previously installed in the ROM 2002or the storage unit 2008.

The program executed by the computer may be a program in whichprocessing is performed in a time-series manner along the orderdescribed in the present specification, or processing is performed inparallel or at necessary timing, for example, when called.

In the present specification, the system represents the entire apparatusthat includes a plurality of apparatuses.

The effects described in the present specification are merely examples,and the effects of the present technology are not limited thereto. Thepresent technology may have another effect.

The embodiment of the present technology is not limited to theembodiment described above. Various modifications may be made withoutdeparting from the gist of the present technology.

The present technology may have the following configuration.

(1)

A medical system including a medical imaging device, and

an image processing apparatus for processing an image captured by themedical imaging device. The image processing apparatus includes:circuitry configured to

acquire a special light image from the image captured by the medicalimaging device, the special light image having information limited to aspecific wavelength band,

generate depth information at a predetermined position of a patientusing the special light image, and detect a structural relationshipusing the depth information.

(2)

The medical system according to (2), wherein the medical imaging deviceis an endoscope or a microscope.

(3)

The medical system according to (1) or (2), wherein the structuralrelationship includes a bleeding position, a transparent membranethickness, a mist resulting from surgery, and/or an overlapping state ofblood vessels.

(4)

The medical system according to (1) or (2), wherein the circuitry isfurther configured to

acquire a normal light image having information limited to a white lightwavelength band, the specific wavelength band corresponds to infraredlight, and detect a bleeding position based on additional depthinformation generated using both the special light image and the normallight image.

(5)

The medical system according to (1) or (2), wherein the circuitry isfurther configured to generate an image in which a predetermined mark ortext representing the bleeding point is superimposed on the speciallight image or the normal light image.

(6)

The medical system according to (1) or (2), wherein the circuitry isfurther configured to acquire a normal light image having informationlimited to a white light wavelength band, the specific wavelength bandcorresponding to polarized light, and detect a membrane thickness of atransparent membrane based on additional depth information generatedusing both the special light image and the normal light image.

(7)

The medical system according to (6), wherein the circuitry is furtherconfigured to generate an image in which a numerical value indicatingthe membrane thickness of the transparent membrane or a gradation imagecorresponding to the membrane thickness, is superimposed on the speciallight image or the normal light image.

(8)

The medical system according to (1) or (2), wherein the circuitry isfurther configured to acquire a normal light image having informationlimited to a white light wavelength band, the specific wavelength bandcorresponding to infrared light, and detect a mist resulting fromsurgery based on additional depth information generated using both thespecial light image and the normal light image.

(9)

The medical system according to (8), wherein the circuitry is furtherconfigured to generate an image in which alarm display that notifies thepresence of the mist or a color that represents the mist, issuperimposed on the special light image or the normal light image.

(10)

The medical system according to (9), wherein the circuitry is furtherconfigured to instruct display of the special light image or generatesan image via image processing that removes the mist resulting from thesurgery.

(11)

The medical system according to (1) or (2), wherein the specificwavelength band includes a first wavelength and a second wavelengthdifferent from the first wavelength, and the circuitry is furtherconfigured to detect positional information in a depth direction of ablood vessel based on additional depth information generated using afirst special light image having information limited to the firstwavelength and a second special light image having information limitedto the second wavelength.

(12)

The medical system according to (12), wherein the circuitry is furtherconfigured to generate an image in which a reflectance of an arterylocated at a deeper position than a vein is corrected using thepositional information in the depth direction of the blood vessel.

(13)

The medical system according to (1) or (2), wherein the specificwavelength band corresponds to excitation light used in excitation lightobservation after indocyanine green (ICG) injection, the circuitry isfurther configured to generate depth information at the predeterminedposition from a depth image obtained by the excitation light, and thecircuitry is further configured to detect a positional relationshipbetween overlapping blood vessels, a blood vessel connected to a cancertissue, or a region having metastatic potential of cancer from the depthinformation.

(14)

The medical system according to (13), wherein the circuitry is furtherconfigured to generate an image in which overlapping blood vessels aredisplayed with different brightnesses or colors or an image, in which aregion having metastatic potential of cancer, is highlighted.

(15)

The medical system according to (1) or (2), wherein the specificwavelength band corresponds to blue light, circuitry is furtherconfigured to generate depth information at the predetermined positionfrom a depth image obtained by the blue light, and circuitry is furtherconfigured to estimate the size in a planar direction of a tumor from animage obtained by the blue light and detect the volume of the tumor bymultiplying a value of the estimated size by the depth information.

(16)

The medical system according to (15), wherein the circuitry is furtherconfigured to generate an image in which the tumor having volume above apredetermined threshold is highlighted.

(17)

The medical system according to (1) or (2), wherein the circuitry isfurther configured to acquire a normal light image having informationlimited to a white light wavelength band, wherein circuitry is furtherconfigured to generate a special light image having information limitedto the specific wavelength band from the normal light image.

(18)

The medical system according to (1) or (2), further including a speciallight source configured to generate the special light for illuminatingthe predetermined position of the patient.

(19)

The medical system according to (1) or (2), wherein the depthinformation is generated with respect to a direction into a body of thepatient or with respect to a direction away from the body of the patientdepending on the structural relationship.

(20)

An image processing method including acquiring a special light image,from the image captured by the medical imaging device, the special lightimage having information limited to a specific wavelength band;generating depth information at a predetermined position of a patientusing the special light image; and detecting a structural relationshipusing the depth information.

(21)

A non-transitory computer readable medium having stored thereon aprogram that when executed by a computer causes the computer to executeprocessing, the processing including acquiring a special light image,from the image captured by the medical imaging device, the special lightimage having information limited to a specific wavelength band;generating depth information at a predetermined position of a patientusing the special light image; and

detecting a structural relationship using the depth information.

(22)

An image processing apparatus for processing an image captured by themedical imaging device, the apparatus including circuitry configured toacquire a special light image from the image captured by the medicalimaging device, the special light image having information limited to aspecific wavelength band, generate depth information at a predeterminedposition of a patient using the special light image, and detect astructural relationship using the depth information.

REFERENCE SIGNS LIST

-   -   101 Image processing apparatus    -   102 Imaging unit    -   103 Display unit    -   121 Normal light image acquisition unit    -   122 Special light image acquisition unit    -   123 3D image acquisition unit    -   124 Depth information generation unit    -   125 Target region detection unit    -   126 Image generation unit    -   141 Normal light image capture unit    -   142 Special light image capture unit    -   143 3D image capture unit    -   222 Special light image acquisition unit    -   321 Normal light image acquisition unit    -   322 Special light image acquisition unit    -   323 3D image acquisition unit    -   341 Normal light 3D image capture unit    -   342 Special light 3D image capture unit

1. A medical system comprising: a medical imaging device; and an imageprocessing apparatus for processing an image captured by the medicalimaging device, comprising: circuitry configured to acquire a speciallight image from the image captured by the medical imaging device, thespecial light image having information limited to a specific wavelengthband, generate depth information at a predetermined position of apatient using the special light image, and detect a structuralrelationship using the depth information.
 2. The medical systemaccording to claim 1, wherein the medical imaging device is an endoscopeor a microscope.
 3. The medical system according to claim 1, wherein thestructural relationship includes a bleeding position, a transparentmembrane thickness, a mist resulting from surgery, and/or an overlappingstate of blood vessels.
 4. The medical system according to claim 1,wherein the circuitry is further configured to acquire a normal lightimage having information limited to a white light wavelength band, thespecific wavelength band corresponds to infrared light, and detect ableeding position based on additional depth information generated usingboth the special light image and the normal light image.
 5. The medicalsystem according to claim 4, wherein the circuitry is further configuredto generate an image in which a predetermined mark or text representingthe bleeding point is superimposed on the special light image or thenormal light image.
 6. The medical system according to claim 1, whereinthe circuitry is further configured to acquire a normal light imagehaving information limited to a white light wavelength band, thespecific wavelength band corresponding to polarized light, and detect amembrane thickness of a transparent membrane based on additional depthinformation generated using both the special light image and the normallight image.
 7. The medical system according to claim 6, wherein thecircuitry is further configured to generate an image in which anumerical value indicating the membrane thickness of the transparentmembrane or a gradation image corresponding to the membrane thickness,is superimposed on the special light image or the normal light image. 8.The medical system according to claim 1, wherein the circuitry isfurther configured to acquire a normal light image having informationlimited to a white light wavelength band, the specific wavelength bandcorresponding to infrared light, and detect a mist resulting fromsurgery based on additional depth information generated using both thespecial light image and the normal light image.
 9. The medical systemaccording to claim 8, wherein the circuitry is further configured togenerate an image in which alarm display that notifies the presence ofthe mist or a color that represents the mist, is superimposed on thespecial light image or the normal light image.
 10. The medical systemaccording to claim 9, wherein the circuitry is further configured toinstruct display of the special light image or generates an image viaimage processing that removes the mist resulting from the surgery. 11.The medical system according to claim 1, wherein the specific wavelengthband includes a first wavelength and a second wavelength different fromthe first wavelength, and the circuitry is further configured to detectpositional information in a depth direction of a blood vessel based onadditional depth information generated using a first special light imagehaving information limited to the first wavelength and a second speciallight image having information limited to the second wavelength.
 12. Themedical system according to claim 11, wherein the circuitry is furtherconfigured to generate an image in which a reflectance of an arterylocated at a deeper position than a vein is corrected using thepositional information in the depth direction of the blood vessel. 13.The medical system according to claim 1, wherein the specific wavelengthband corresponds to excitation light used in excitation lightobservation after indocyanine green (ICG) injection, the circuitry isfurther configured to generate depth information at the predeterminedposition from a depth image obtained by the excitation light, and thecircuitry is further configured to detect a positional relationshipbetween overlapping blood vessels, a blood vessel connected to a cancertissue, or a region having metastatic potential of cancer from the depthinformation.
 14. The medical system according to claim 13, wherein thecircuitry is further configured to generate an image in whichoverlapping blood vessels are displayed with different brightnesses orcolors or an image, in which a region having metastatic potential ofcancer, is highlighted.
 15. The medical system according to claim 1,wherein the specific wavelength band corresponds to blue light,circuitry is further configured to generate depth information at thepredetermined position from a depth image obtained by the blue light,and circuitry is further configured to estimate the size in a planardirection of a tumor from an image obtained by the blue light and detectthe volume of the tumor by multiplying a value of the estimated size bythe depth information.
 16. The medical system according to claim 15,wherein the circuitry is further configured to generate an image inwhich the tumor having volume above a predetermined threshold ishighlighted.
 17. The medical system according to claim 1, wherein thecircuitry is further configured to acquire a normal light image havinginformation limited to a white light wavelength band, wherein circuitryis further configured to generate a special light image havinginformation limited to the specific wavelength band from the normallight image.
 18. The medical system according to claim 1, furthercomprising: a special light source configured to generate the speciallight for illuminating the predetermined position of the patient. 19.The medical system according to claim 1, wherein the depth informationis generated with respect to a direction into a body of the patient orwith respect to a direction away from the body of the patient dependingon the structural relationship.
 20. An image processing methodcomprising: acquiring a special light image, from the image captured bythe medical imaging device, the special light image having informationlimited to a specific wavelength band; generating depth information at apredetermined position of a patient using the special light image; anddetecting a structural relationship using the depth information.
 21. Anon-transitory computer readable medium having stored thereon a programthat when executed by a computer causes the computer to executeprocessing, the processing comprising: acquiring a special light image,from the image captured by the medical imaging device, the special lightimage having information limited to a specific wavelength band;generating depth information at a predetermined position of a patientusing the special light image; and detecting a structural relationshipusing the depth information.
 22. An image processing apparatus forprocessing an image captured by the medical imaging device, comprising:circuitry configured to acquire a special light image from the imagecaptured by the medical imaging device, the special light image havinginformation limited to a specific wavelength band, generate depthinformation at a predetermined position of a patient using the speciallight image, and detect a structural relationship using the depthinformation.